Image processing apparatus, image processing method, and program

ABSTRACT

Provided are an apparatus and method for executing sensitivity difference correction processing of an image signal, which is generated by a single plate-type image sensor through a color filter. The sensitivity difference correction is executed for Gr and Gb signals included the image signal, for example, an RGB signal, which is generated by the single plate-type image sensor through the color filter. A pixel value of a color filter unit which has the same color as a correction target pixel and is present in surroundings of the correction target pixel is acquired. An additional value is calculated by adding a difference between weighted mean pixel values “a” and “b” of two kinds of pixel groups “A” and “B” classified according to positions of pixels to the pixel value of the correction target pixel in which the weighted mean values correspond to distances of the pixel groups from the correction target pixel. A mean value of the pixel value of the correction target pixel and the additional value is calculated as a corrected pixel value of the correction target pixel.

TECHNICAL FIELD

The invention relates to an image processing apparatus, an imageprocessing method, and a program. In particular, the invention relatesto an image processing apparatus, an image processing method, and aprogram that performs signal processing on an output of a singleplate-type image sensor.

BACKGROUND ART

When a single plate-type solid-state image sensor is used as an imagesensor of an imaging apparatus, only a single spectral sensitivity isobtained. Therefore, generally, color imaging is performed by arrangingcolor filters of different colors such as R, G, and B on image sensorscorresponding to respective pixels. In this method, only one color (forexample, any one of R, G, and B) is obtained with each pixel.Accordingly, a mosaic-like image based on color is generated.

Specifically, only color information, R or G or B or the like, isacquired for each pixel according to the pattern of filters. This imageis called a so-called mosaic image. In order to obtain a color imagefrom the mosaic image, it is necessary to obtain the color informationof every color for each of all the pixels.

A color image can be generated by calculating color information of allcolors (for example, all the RGB) corresponding to each of all pixels byinterpolating color information obtained from surrounding pixels of eachpixel. This interpolation processing is called demosaic processing.

For example, an example of the color filters used for an imagingapparatus is illustrated in FIG. 1(1). This array is called Bayerpattern, and transmits light (R, G, or B) having a specific wavelengthcomponent in units of a pixel. In the Bayer pattern, a minimum unitconsists of four pixels which include two filters to transmit a green(G), one filter to transmit a blue (B), and one filter to transmit a red(R).

With the miniaturization of the image sensor, the sensor becomes easilyaffected by a minute difference in pixel structure. Therefore, itbecomes apparent that even the pixels having the same spectralcharacteristic (for example, G pixels in the Bayer pattern) aredifferent in sensitivity from pixel to pixel due to the slightdifference in the structure.

For example, as illustrated in FIG. 1(2), the G pixels include G pixelsin an R line (hereinafter, referred to as Gr pixels) and G pixels in a Bline (hereinafter, referred to as Gb pixels). Although the Gr pixelshave G filters having the same spectral characteristic as those of theGb pixels, there might be sensitivity differences because of the slightstructural differences.

When the above-mentioned demosaic processing is performed on an imageimaged by the image sensors having the sensitivity differences, aportion having small differences in brightness, which may be originallydetermined to be a flat portion, is erroneously determined to be an edgeportion due to the difference in DC component between the Gb pixels andthe Gr pixels. As a result, an error occurs in selecting surroundingpixels used to determine a pixel value of a specific pixel, so that aplurality of interpolation values is mixed irregularly. This is likelyto result in generation of an artifact that stands out very much.Therefore, it is necessary to perform correction processing on thesensitivity difference before the demosaic processing is performed. Thedemosaic processing is described, for example, in Patent Document 1(Japanese Patent No. 2931520).

When the sensitivity differences have the same tendency over the entirescreen, the correction may be performed by adjusting the level and/orthe offset. That is, since the correction is performed such that thesensitivity of the G pixels in a R line (Gr pixels) matches thesensitivity of the G pixels in a B line (Gb pixels), the following canbe estimated with use of coefficients A and B.

[Formula 1]

Gb=Gr×A+B  (Expression 1)

In the above expression (Expression 1), data that shows a bar above Gbindicates a pixel value obtained by correcting the sensitivity of the Gpixels in an R line (Gr pixels) so as to match the sensitivity of the Gbpixels. The symbol “-” (bar) written above Gb or the like in expressionsis written in the form of Gb(-) in the specification.

When the sensitivity differences have the same tendency over the entirescreen, it is effective to use a correction value obtained by using theabove expression (Expression 1). However, causes of generation of thesensitivity differences at the positions of respective pixels includevarious factors such as a pixel structure and an angle of incidentlight. Therefore, the sensitivity varies from pixel to pixel (forexample, an upper side and a lower side of a screen). Moreover, even thesame pixel changes in sensitivity due to the influence of the apertureof a lens or the like.

A method of absorbing the level difference according to the pixelposition is also proposed. The method measures the difference in thesensitivity of each area, and absorbs the sensitivity difference byperforming the correction processing based on the gain and the offset.For example, when a horizontal distance from the center is assumed to bex and a vertical distance is assumed to be y, a correction coefficientfor each pixel can be approximately calculated by using a correctionfunction f(x, y) and a correction function g(x, y) calculated from thesensitivity difference of each area, and the correction can be performedas follows.

[Formula 2]

Gb=Gr×f(x,y)+g(x,y)  (Expression 2)

However, this method achieves only a rough correction for each area.Accordingly, the sensitivity difference between fine areas cannot beabsorbed. In addition, since the sensitivity also depends on opticalcharacteristics such as an aperture and a zoom state of a lens, a greatdeal of labor and time is required to measure the f(x, y) and/or theg(x, y).

There is also a technique to absorb the sensitivity difference by usingonly information on adjacent pixels. In Patent Document 2 (JapanesePatent Application Laid-open (JP-A) No. 2005-160044), image processingis performed when performing the demosaic processing on an image of fourcolors obtained by using a color filter for transmitting an emerald (E)in addition to filters for transmitting R, G, and B as illustrated inFIG. 1(3) by using the fact that spectral characteristics of colorfilters of G and E are similar to each other. By estimating E pixels atthe positions of G pixels and estimating G pixels at the positions of Epixels, an image illustrated in FIG. 1(4) can be produced.

For the image arrayed as illustrated in FIG. 1(4), it becomes possibleto perform demosaic processing similar to the demosaic processing whichis applied to the Bayer pattern (FIG. 1(1)). However, although thespectral characteristics and/or the sensitivities of the G filter andthe E filter are different as illustrated in FIG. 2, since the spectralcharacteristics thereof partially overlap, there is a strong correlationbetween the G pixel and the E pixel. Accordingly, estimation ofregression analysis can be used to estimate the E pixels at thepositions of the G pixels or to estimate the G pixels at the positionsof the E pixels.

The technique which estimates G pixels at the positions of E pixels isillustrated as an example. The weighted mean mE and mG of adjacent Epixels is calculated as follows.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 3} \right\rbrack & \; \\{{mE} = \frac{\sum\limits_{i}^{\;}\left( {E_{i} \times C_{i}} \right)}{\sum\limits_{i}^{\;}C_{i}}} & \left( {{Expression}\mspace{14mu} 3} \right) \\{{mG} = \frac{\sum\limits_{j}^{\;}\left( {G_{j} \times C_{j}} \right)}{\sum\limits_{j}^{\;}C_{j}}} & \left( {{Expression}\mspace{14mu} 4} \right)\end{matrix}$

In the above expressions (Expression 3) and (Expression 4), i representsa pixel number of a certain surrounding pixel, Ei represents a pixelvalue of an E pixel corresponding to the number, and Ci represents aweighting factor corresponding to the distance from a center pixel. jrepresents a pixel number of another certain surrounding pixel, Gjrepresents a pixel value of a G pixel corresponding to the number, andCj represents a weighting factor corresponding to the distance from thecenter pixel.

Dispersion VGG of the adjacent G pixels and covariance VEG of the Gpixels and the E pixels are calculated considering the difference in thespectral characteristic between the E pixel and the G pixel illustratedin FIG. 2, and an estimation value of the E pixel is estimated asfollows.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 4} \right\rbrack & \; \\{\overset{\_}{E} = {{\frac{V_{EG}}{V_{GG}} \times \left( {G - {mG}} \right)} + {mE}}} & \left( {{Expression}\mspace{14mu} 5} \right)\end{matrix}$

In the above-mentioned (Expression 5), it is necessary to performcalculations of the dispersion and the covariance, and the calculationamount of these operations is very large. Accordingly, in some cases,such estimation is practically performed by using an operation lighterthan (Expression 5) as described below.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 5} \right\rbrack & \; \\{\overset{\_}{E} = {\frac{G}{mG} \times {mE}}} & \left( {{Expression}\mspace{14mu} 6} \right)\end{matrix}$

However, it is also understood that this expression (Expression 6)requires multiplication and division operations. Furthermore, it isunderstood that, when it is achieved with a circuit, it costs a lot.

The technique disclosed in Patent Document 2 (JP-A No. 2005-160044) canbe used not only to perform processing on R, G, B, and E illustrated inFIG. 1(4) but also to perform correction by estimating the sensitivitydifference between the Gb pixel and the Gr pixel in the Bayer patternillustrated in FIG. 1(2). However, a great amount of calculations isnecessary to calculate the dispersion and the covariance in theabove-mentioned (Expression 5), and an amount of calculation is alsolarge in a simpler expression (Expression 6) because it includesdivisions.

The spectral characteristics of the G filter and the E filter aredifferent as illustrated in FIG. 2 in the case of the array of R, G, B,and E illustrated in FIG. 1(4). However, in the Bayer patternillustrated in FIG. 1(2), although the filter characteristics of the Gbfilter and the Gr filter are affected by color mixture, pixel structure,and incident light, the filter characteristics are very similar to eachother as illustrated in FIG. 3. Therefore, it is anticipated that thereis a correlation between them which is stronger than the correlationbetween the G pixel and the E pixel, and the correction can be achievedwith a smaller amount of operations.

CITATION LIST Patent Documents

-   Patent Document 1: Japanese Patent No. 2931520-   Patent Document 2: Japanese Patent Application Laid-Open No.    2005-160044

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

The invention is made in view of the above problems and is intended toprovide an image processing apparatus, an image processing method, and aprogram which can perform correction processing on a mosaic image imagedby using a specific color filter for a single plate-type image sensorwith a smaller amount of operations and with high accuracy and whichachieves a high-quality color image.

Solutions to Problems

A first aspect of the present invention is an image processing apparatusincluding:

a sensitivity difference correction processing unit that receives aphotoelectric conversion signal and executes sensitivity differencecorrection processing, the photoelectric conversion signal being outputby an image sensor based on incident light being incident through acolor filter,

wherein the sensitivity difference positive processing unit: acquires apixel value of a color filter unit which has a same color as acorrection target pixel and is present in surroundings of the correctiontarget pixel; calculates an additional value by adding a differencebetween weighted mean pixel values “a” and “b” of two kinds of pixelgroups “A” and “B” classified according to positions of the pixels tothe pixel value of the correction target pixel, the weighted mean pixelvalues corresponding to distances of the pixel groups from thecorrection target pixel; and calculates a mean value of the pixel valueof the correction target pixel and the additional value as a correctedpixel value of the correction target pixel.

Furthermore, according to an embodiment of the image processingapparatus of the present invention, the color filter is arrayed in aBayer pattern, and the sensitivity difference correction processing unithas a configuration which a sensitivity difference between an Gr pixeland a Gb pixel included in an image which is imaged through the colorfilter of the Bayer pattern is corrected.

Furthermore, according to an embodiment of the image processingapparatus of the present invention, the sensitivity differencecorrection processing unit: calculates, when performing correctionprocessing on the Gr pixel and the Gb pixel, weighted pixel value meansmGr and mGb corresponding to distances of the Gr pixel and the Gb pixel,which are adjacent to the correction target pixel, from the correctiontarget pixel; calculates an additional value by adding a differencebetween the mGr and the mGb to the pixel value of the correction targetpixel; and calculates the mean value of the pixel value of thecorrection target pixel and the additional value as the corrected pixelvalue of the correction target pixel.

Furthermore, according to an embodiment of the image processingapparatus of the present invention, the sensitivity differencecorrection processing unit: calculates a final corrected pixel value byreducing a contribution level of the additional value when an adjacentimage area including the correction target pixel is an edge area; andcalculates the final corrected pixel value by increasing thecontribution level of the additional value when the adjacent image areaincluding the correction target pixel is a flat area.

Furthermore, according to an embodiment of the image processingapparatus of the present invention, the sensitivity differencecorrection processing unit has a configuration in which demosaicprocessing is executed.

Furthermore, according to an embodiment of the image processingapparatus of the present invention, the sensitivity differencecorrection processing unit has a configuration in which demosaicprocessing of determining a pixel value of a processing target pixel isperformed based on a pixel value of a reference pixel by determining thecontribution level of the pixel value of the reference pixel adjacent tothe processing target pixel according to a result of the edge directiondetermination of an image area including the processing target pixel ofthe demosaic processing.

Furthermore, according to an embodiment of the image processingapparatus of the present invention, the sensitivity differencecorrection processing unit has a configuration in which demosaicprocessing of determining the pixel value of the processing target pixelis performed by setting the contribution level of the pixel value of thereference pixel in a direction where a change in pixel value is small toa high level, according to a result of the edge direction determinationof the image area including the processing target pixel of the demosaicprocessing.

A second aspect of the present invention is an image processingapparatus including:

an imaging device; and

the sensitivity difference correction processing unit.

A third aspect of the present invention is an image processing methodexecuted in an image processing apparatus, the method including:

by a sensitivity difference correction processing unit, receiving aphotoelectric conversion signal that is output by an image sensor basedon incident light being incident through a color filter, and executingsensitivity difference correction processing, wherein

the sensitivity difference positive processing executes: acquiring apixel value of a color filter unit which has a same color as acorrection target pixel and is present in surroundings of the correctiontarget pixel; calculating an additional value by adding a differencebetween weighted mean pixel values “a” and “b” of two kinds of pixelgroups “A” and “B” classified according to positions of pixels, theweighted pixel value means corresponding to distances of the pixelgroups from the correction target pixel, to a pixel value of thecorrection target pixel; and calculating a mean value of the pixel valueof the correction target pixel and the additional value as a correctedpixel value of the correction target pixel.

A fourth aspect of the present invention is an program that causes imageprocessing to be executed in an image processing apparatus and causes asensitivity difference correction processing unit to receive aphotoelectric conversion signal that is output by an image sensor basedon incident light being incident through a color filter, and to executesensitivity difference correction processing, wherein

the sensitivity difference positive processing executes: acquiring apixel value of a color filter unit which has a same color as acorrection target pixel and is present in surroundings of the correctiontarget pixel; calculating an additional value by adding a differencebetween weighted mean pixel values “a” and “b” of two kinds of pixelgroups “A” and “B” classified according to positions of pixels, theweighted pixel value means corresponding to distances of the pixelgroups from the correction target pixel, to a pixel value of thecorrection target pixel; and calculating a mean value of the pixel valueof the correction target pixel and the additional value as a correctedpixel value of the correction target pixel.

The program of the invention is, for example, a program that can beprovided by a communication medium or a recording medium which isprovided, for example, to an image processing apparatus or a computersystem, which can execute various program codes, in a computer-readableformat. By providing such a program in a computer-readable format, theprocessing performed according to the program can be implemented in theimage processing apparatus or the computer system.

Other objects, features, and advantages of the invention will becomeapparent in light of a more detailed description based on embodiments ofthe invention to be described later and the attached drawings. The term“system” in this specification means a configuration of a logical set ofa plurality of devices, and each device in the configuration is notnecessarily provided in the same casing.

Effects of the Invention

According to a configuration of one embodiment of the invention,provided are an apparatus and a method that execute sensitivitycorrection processing of an image signal generated by a singleplate-type image sensor through a color filter. Specifically, thesensitivity difference correction of Gr and Gb signals included in animage signal, for example, an RGB signal, generated by a singleplate-type image sensor through a color filter. Pixel values of colorfilter units having a color the same as that of a correction targetpixel and being provided around the correction target pixel areacquired. A difference between weighted mean pixel values “a” and “b” oftwo kinds of pixel groups “A” and “B” classified according to positionsof pixels is added to a pixel value of the correction target pixel tocalculate an additional value in which the weighted mean pixel valuesare based on the distance from the correction target pixel. A mean valueof the pixel value of the correction target pixel and the additionalvalue is calculated as a corrected pixel value of the correctioninterest pixel. According to this processing, the correction processingcan be performed with a simple operation and with high accuracy.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram that describes a Bayer pattern, which is a colorarray, used for general color filters, and a problem thereof.

FIG. 2 is a diagram that describes spectral characteristics andsensitivities of a G filter and an E filter.

FIG. 3 is a diagram that describes filter characteristics of a Gb filterand a Gr filter.

FIG. 4 is a diagram that describes a configuration example of an imageprocessing apparatus of the invention.

FIG. 5 is a diagram that describes a configuration example of a DSPblock 103 of the image processing apparatus of the invention.

FIG. 6 is a diagram that describes details of a sensitivity differencecorrection processing unit 111 of the DSP block 103 of the imageprocessing apparatus of the invention.

FIG. 7 is a diagram that describes processing in which a mean value ofGb and Gr obtained by correcting the sensitivity difference is output toa position of a Gr pixel and a position of a Gb pixel by correctionprocessing of the image processing apparatus of the invention.

FIG. 8 is a diagram that describes an example of an image signal havinga monochrome edge of a vertical stripe.

FIG. 9 is a diagram that describes an example of a function β(x) thatyields different values for an edge area and a flat area respectively.

FIG. 10 is a diagram that describes another example of the function β(x)that yields different values for an edge area and a flat arearespectively.

FIG. 11 is a diagram that describes a configuration example of a DSPblock 103 in a sixth embodiment.

FIG. 12 is a diagram that describes a sensitivity difference correctionand demosaic processing unit 131 of the DSP block 103 in the sixthembodiment.

FIG. 13 is a diagram that describes raster scan processing.

FIG. 14 is a diagram that describes an example of calculating agradient.

FIG. 15 is a diagram that describes a configuration example of acorrelation processing unit 149 in the sensitivity difference correctionand demosaic processing unit 131 of the DSP block 103 in the sixthembodiment.

FIG. 16 is a diagram that describes processing executed by a G highfrequency generating unit 151 of the correlation processing unit 149.

FIG. 17 is a diagram that describes the sensitivity differencecorrection and demosaic processing unit 131 in a seventh embodiment.

FIG. 18 is a diagram that describes an example of an advantage obtainedby the processing of the invention.

FIG. 19 is a diagram that describes a configuration example of hardwareof the image processing apparatus of the invention.

MODE FOR CARRYING OUT THE INVENTION

Hereafter, an image processing apparatus, an image processing method,and a program of the invention are described with reference to thedrawings. A description is made in the following order:

1. Regarding a configuration of an image processing apparatus andsensitivity difference correction processing according to a firstembodiment of the invention;

2. Regarding an example of sensitivity difference correction processingin which an edge portion is considered (second embodiment);

3. Regarding an example of sensitivity difference correction processingin which an edge portion is considered (third embodiment);

4. Regarding an example of sensitivity difference correction processingin which an edge portion is considered (fourth embodiment);

5. Regarding an example of sensitivity difference correction processingin which an edge portion is considered (fifth embodiment);

6. Regarding an embodiment (sixth embodiment) in which sensitivitydifference correction processing and demosaic processing are combined;

7. Regarding an embodiment (seventh embodiment) in which sensitivitydifference correction processing and demosaic processing are combined;and

8. Regarding a hardware configuration example of an image processingapparatus.

1. Regarding a Configuration of an Image Processing Apparatus andSensitivity Difference Correction Processing According to a FirstEmbodiment of the Invention

A configuration example of an image processing apparatus of theinvention is described with reference to FIG. 4. FIG. 4 is a blockdiagram that illustrates an example of a configuration of an imagingapparatus that is one embodiment of the image processing apparatus ofthe invention. The image processing apparatus of the invention may be anapparatus that can execute demosaic processing by receiving a mosaicimage imaged by using a specific color filter for a single plate-typeimage sensor. The image processing apparatus of the invention is notlimited to an imaging apparatus and can be implemented by a device suchas a PC. Hereinbelow, an example of an imaging apparatus is described asan example of the image processing apparatus of the invention.

In an imaging apparatus 100 illustrated in FIG. 4, light incidentthrough an optical lens 101 is incident on an imaging device 102configured by a CCD, a CMOS image sensor, or the like. The imagingdevice 102 generates image data (a photoelectric conversion signal),based on the light, which is input to a single plate-type image sensorthrough a color filter, and inputs the image data to a DSP block 103.

The image data (the photoelectric conversion signal) is input to the DSPblock 103, and is then subjected to signal processing for an outputsignal here. As a result, output image 104 is output.

An example of the configuration of the DSP block 103 is described withreference to FIG. 5. An image of the Bayer pattern illustrated in FIG.1(1) is output from the imaging device 102 illustrated in FIG. 4 and isinput to the DSP block 103. However, this input data has the sensitivitydifferences according to the positions of G. That is, the input databecomes mosaic data that corresponds to the array (R, Gb, Gr, B)illustrated in FIG. 1(2). The positions of Gb and Gr pixels arepositions where the same G filters are set. However, since the positionsare different, they are sensitive to beams of slightly differentwavelengths. That is, they have a sensitivity difference.

In a sensitivity difference correction processing unit 111 of the DSPblock 103, the sensitivity difference between the Gr pixel and the Gbpixel which has the same optical characteristic in the Bayer patternillustrated in FIG. 1(2) is corrected and an image of the Bayer pattern(RGB) illustrated in FIG. 1(1) is output. That is, a photoelectricconversion signal generated based on the light input to the singleplate-type image sensor through the color filter is input andsensitivity difference correction processing is executed.

In a white balance processing unit 112, pixel values of R, G, and Bchannels in an achromatic color area are multiplied by appropriatecoefficients corresponding to respective colors so that the pixel valuesmay become equal to each other, and a white balance-adjusted image isoutput. A demosaic processing unit 113 executes demosaic processing thatsets all RGB pixel values for each pixel position, with respect to amosaic image where only one pixel value out of information of the RGBpixel values is set for each pixel position. Specifically, it performsequalization processing of providing each pixel position with threechannels of R, G, and B by using pixel value information of adjacentpixels and outputs the resultant.

In a gamma correcting unit 114, nonlinear correction processing isperformed so that the brightness and the color saturation of the image,which is output as an output image 104 illustrated in FIG. 4, can becorrectly displayed and the corrected image is output. A YC convertingunit 115 generates and outputs an Y image and a C image by performingmatrix conversion processing on the input three-channel image (imagewith R, G, B channels) and performing processing of limiting bandwidthsof chromatic components.

Details of the sensitivity difference correction processing unit 111 aredescribed with reference to FIG. 6. FIG. 6 is a diagram that illustratesan example of a detailed configuration of the sensitivity differencecorrection processing unit 111 (see FIG. 5) of the DSP block 103 of theimage processing apparatus 100 of the invention illustrated in FIG. 5.

A Gr mean value calculating unit 121 is a block to undergo a weightedmean of the pixel values of adjacent Gr pixels of an interest pixel. Theinterest pixel is a pixel to be subjected to pixel-value correction andsetting processing. For all the pixels which form the image, the pixelsare sequentially selected one by one and the processing is performed.When the weighted mean of Gr is assumed to be mGr, the Gr mean valuecalculating unit 121 calculates the mGr based on the followingexpression (Expression 7).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 6} \right\rbrack & \; \\{{mGr} = \frac{\sum\limits_{i}^{\;}\left( {{Gr}_{i} \times C_{i}} \right)}{\sum\limits_{i}^{\;}C_{i}}} & \left( {{Expression}\mspace{14mu} 7} \right)\end{matrix}$

The weighted mean of Gr, mGr, can be calculated by the above-mentioned(Expression 7). In the above expression (Expression 7), i represents apixel number of a certain surrounding pixel, Gri represents a pixelvalue of the Gr pixel corresponding to the pixel number i, and Cirepresents a weighting factor corresponding to the distance from acenter pixel.

A Gb mean value calculating unit 122 is a block to undergo a weightedmean of the pixel values of the Gb adjacent to the interest pixel. Whena weighted mean of the Gb is assumed to be mGb, the mGb can becalculated based on the following expression (Expression 8).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 7} \right\rbrack & \; \\{{mGb} = \frac{\sum\limits_{j}^{\;}\left( {{Gb}_{j} \times C_{j}} \right)}{\sum\limits_{j}^{\;}C_{j}}} & \left( {{Expression}\mspace{14mu} 8} \right)\end{matrix}$

In the above expression (Expression 8), j represents a pixel number of acertain surrounding pixel, Gbj represents a pixel value of the Gb pixelcorresponding to the pixel number j, and Cj represents a weightingfactor corresponding to the distance from the center pixel.

In a sensitivity difference correcting unit 123, the sensitivitydifference of the interest pixel is corrected by using the Gb or the Grwhich is the interest pixel, the weighted mean mGr of the Gr obtained bythe Gr mean value calculating unit 121, and the weighted mean mGb of theGb obtained by the Gb mean value calculating unit 122.

As illustrated in FIG. 7, a mean value of sensitivity-corrected Gb andGr is calculated and output to the positions of the Gb pixels and the Grpixels by the correction processing.

A method of calculating estimation values of the Gb pixels and the Grpixels is shown here. As previously described with reference to FIG. 3,the characteristics of the Gb filter and the Gr filter are the same, andonly the sensitivities thereof are slightly different due to theinfluence of the pixel structure and the incident light. Therefore, theimage can be sufficiently corrected by removing the sensitivitydifference at a low frequency by using the weighted mean mGr of theadjacent Gr pixels and the weighted mean mGb of the Gb pixels.

When a Gb pixel is the interest pixel as a correction target, it isnecessary to estimate a value of the Gr pixel at the position of theinterest pixel. The value of the Gr pixel at the position of theinterest pixel Gb means the pixel value when the Gb pixel, which is theinterest pixel, has the same sensitivity as the Gr pixel being presentin the surroundings. The estimation value Gr(-) of the Gr pixel at theposition of the interest pixel can be calculated as follows. Inaddition, as described above, the symbol “-(bar)” above Gr or the likein expressions is written in the form of Gr(-) in the specification.

[Formula 8]

Gr=Gb+(mGr−mGb)  (Expression 9)

In the above-mentioned (Expression 9), Gb is the pixel value of theinterest pixel, and mGr and mGb are the weighted mean values of Gr andGb adjacent to the interest pixel which are calculated based on thepreviously described (Expression 7) and (Expression 8).

When the Gr pixel is an interest pixel as a correction processingtarget, it is necessary to estimate the Gb pixel at the position of theinterest pixel. The value of the Gb pixel at the position of theinterest Gr pixel means a pixel value when the Gr pixel, which is aninterest pixel, has the same sensitivity as the surrounding Gb pixel.Estimation value Gb(-) of the Gb pixel at the position of the interestpixel can be calculated as follows.

[Formula 9]

Gb=Gr+(mGb−mGr)  (Expression 10)

In the above-mentioned (Expression 10), Gr is the pixel value of theinterest pixel. mGr and mGb are the weighted mean values of Gr and Gbadjacent to the interest pixel which are calculated based on thepreviously described (Expression 7) and (Expression 8).

By using the above-mentioned (Expression 9) and (Expression 10), asillustrated in FIG. 7, sensitivity-corrected G pixel values can be setfor the positions of all the G pixels (original Gb and Gr). Thesensitivity-corrected G pixel value (G(-)) is calculated as follows.When a Gb pixel is the interest pixel as a correction target, the pixelvalue of the interest pixel becomes an arithmetic mean value of theoriginal pixel value of the Gb pixel and the estimation value Gr(-) ofthe Gr pixel calculated by the above-mentioned (Expression 9). When a Grpixel is the interest pixel as a correction target, the pixel value ofthe interest pixel becomes an arithmetic mean value of the originalpixel value of the Gr pixel and the estimation value Gb(-) of the Gbpixel calculated by the above-mentioned (Expression 10).

A specific expression is shown below. When the interest pixel is Gb, a Gpixel value G(-) which has undergone the sensitivity differencecorrection is calculated by the following (Expression 11).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 10} \right\rbrack & \; \\\begin{matrix}{\overset{\_}{G} = \frac{{Gb} + {\overset{\_}{G}r}}{2}} \\{= {{Gb} + \frac{{mGr} - {mGb}}{2}}}\end{matrix} & \left( {{Expression}\mspace{14mu} 11} \right)\end{matrix}$

Moreover, when the interest pixel is Gr, a G pixel value G(-) which hasundergone the sensitivity difference correction is calculated by thefollowing (Expression 12).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 11} \right\rbrack & \; \\\begin{matrix}{\overset{\_}{G} = \frac{{\overset{\_}{G}b} + {Gr}}{2}} \\{= {{Gr} + \frac{{mGb} - {mGr}}{2}}}\end{matrix} & \left( {{Expression}\mspace{14mu} 12} \right)\end{matrix}$

In the invention, the corrected G pixel value corresponding to theposition of the Gb pixel or the Gr pixel which is obtained by asingle-plate type image sensor is calculated by using theabove-described expressions, that is, from (Expression 7) to (Expression12).

As understood by referring to the above-mentioned (Expression 7) to(Expression 12), the operations of (Expression 7) to (Expression 12) canbe executed only by additions and subtractions. Ci and Cj that aredenominators in (Expression 7) and (Expression 8) are coefficientscorresponding to the distances from the interest pixel and can be freelyset. Therefore, the value of the denominator in (Expression 7) and(Expression 8), which is the sum of Ci and Cj, can be freely set. Whenit is set to the square of the sum, the division can be achieved only bybit shifting. Since the divisions in (Expression 11) and (Expression 12)can be achieved by the bit shifting, all the expressions from(Expression 7) to (Expression 12) can be achieved only by additions andsubtractions. Therefore, a configuration which calculates the value ofthe corrected G pixel at the position of the Gr pixel or the Gb pixelwith a far lighter operation than conventional methods which use thepreviously described (Expression 5) and (Expression 6) can be achieved.

2. Regarding an Example of Sensitivity Difference Correction Processingin which an Edge Portion is Considered Second Embodiment

Next, an example of sensitivity difference correction processing inwhich an edge is considered is described as a second embodiment of theimage processing apparatus of the invention.

As described above, in the image processing apparatus of the invention,the sensitivity difference correction can be performed by using anoperation process to which the previously described (Expression 7) to(Expression 12) are applied. In an image generated by the demosaicprocessing or the like performed based on the G pixel obtained by thecorrection processing, the artifact especially in the flat portiondisappears so that a fairly good output image is obtained in many cases.

However, this correction processing is also accompanied by an aspect inwhich frequency characteristic in the vicinity of the Nyquist frequencyis deteriorated. A case where an image signal having a monochrome edgeof a vertical stripe as illustrated in FIG. 8 is input will be describedas an example.

When the G pixel values, which have undergone the sensitivity correctionusing the previously described (Expression 11) and (Expression 12), areset with respect to the image signal having a monochrome edge of avertical stripe illustrated in FIG. 8, both of the Gb pixel and the Grpixel have an intermediate value between a value of white and a value ofblack, that is, gray. As the result of correction, the G pixels become aflat image. That is, the above-described correction processing is likelyto result in blurry edges.

Hereafter, an embodiment to prevent blurry edges is described as asecond embodiment. An image processing apparatus of the secondembodiment has the configuration which has been described referring toFIGS. 4 to 6 like in first embodiment. However, in the presentembodiment, an expression obtained by modifying arithmetic expressions(Expression 11 and Expression 12) which calculate the pixel values ofthe G pixels, G(-), which have undergone the sensitivity differencecorrection in first embodiment is used to calculate corrected pixelvalues of the G pixels.

In the second embodiment, when an interest pixel is Gb, the G pixelvalue G(-) which has undergone the sensitivity difference correction iscalculated by replacing (Expression 11) which has been described in thefirst embodiment with the following (Expression 13) or (Expression 15)to be described below, and by applying (Expression 13) or (Expression15). Moreover, when the interest pixel is Gr, the G pixel value G(-)which has undergone the sensitivity correction is calculated byreplacing (Expression 12) which has been described in the firstembodiment with the following (Expression 14) or (Expression 16) to bedescribed below, and by applying (Expression 14) or (Expression 16).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 12} \right\rbrack & \; \\{\overset{\_}{G} = {{Gb} + {\alpha \times \left( \frac{{mGr} - {mGb}}{2} \right)}}} & \left( {{Expression}\mspace{14mu} 13} \right) \\{\overset{\_}{G} = {{Gr} + {\alpha \times \left( \frac{{mGb} - {mGr}}{2} \right)}}} & \left( {{Expression}\mspace{14mu} 14} \right)\end{matrix}$

In the above-mentioned (Expression 13) and (Expression 14), α is acoefficient which indicates a correction effect, and is in the range of0≦α≦1. When the coefficient α is set to a small value, the influence ofthe subsequent terms of α in the expression is less, and the correctioneffect becomes weak. Accordingly, the edge characteristic may remain. Onthe other hand, when the coefficient is set to a large value, thecorrection effect becomes strong. As a result, the artifact in the flatportion does not stand out.

In the correction of the pixel value, it is difficult to reduce only theartifact with the edge characteristic being maintained. That is, theartifact in the flat portion and the edge characteristic are in thetrade-off relationship, so that it is difficult to achieve both thecorrection of the artifact in the flat portion and the maintaining ofthe edge portion at the same time.

However, if adaptive processing is performed in which the coefficient αin the above-mentioned (Expression 13) and (Expression 14) is set to 0or a value near 0 for the edge portion and the coefficient α is set to 1or a value near 1 for the flat portion, it is possible to correct theartifact of the flat portion while maintaining the edge portion.

In addition, in order to achieve the correction of the artifact of theflat portion with the edge portion maintained, the above-mentioned(Expression 13) and (Expression 14) are replaced with the following(Expression 15) and (Expression 16).

When the interest pixel is Gb, the G pixel value G(-) which hasundergone the sensitivity difference correction is calculated by usingthe following (Expression 15). Moreover, when the interest pixel is Gr,the G pixel value G(-) which has undergone the sensitivity differencecorrection is calculated by using the following (Expression 16).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 13} \right\rbrack & \; \\{\overset{\_}{G} = {{Gb} + {{\beta \left( \frac{{mGr} - {mGb}}{2} \right)} \times \left( \frac{{mGr} - {mGb}}{2} \right)}}} & \left( {{Expression}\mspace{14mu} 15} \right) \\{\overset{\_}{G} = {{Gr} + {{\beta \left( \frac{{mGb} - {mGr}}{2} \right)} \times \left( \frac{{mGb} - {mGr}}{2} \right)}}} & \left( {{Expression}\mspace{14mu} 16} \right)\end{matrix}$

(Expression 15) and (Expression 16) correspond to the expressionsobtained by substituting the coefficient α in the previously described(Expression 13) and (Expression 14) as follows.

α=β((mGr−mGb)/2), or

α=β((mGb−mGr)/2)

A function β() in the above-mentioned (Expression 15) and (Expression16) is a function defined by following (Expression 17).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 14} \right\rbrack & \; \\{{\beta (x)} = \left\{ \begin{matrix}1 & {{x} < {TH}} \\0 & {otherwise}\end{matrix} \right.} & \left( {{Expression}\mspace{14mu} 17} \right)\end{matrix}$

In the above-mentioned (Expression 17), TH is a threshold parameter toseparate the flat portion and the edge portion from each other. Afunction β(x) defined by the above-mentioned (Expression 17) correspondsto a function illustrated in FIG. 9(1). The function β(x) is a functionthat produces a value different from the edge area and the flat area.

The image processing apparatus of the second embodiment has the sameconfiguration as the image processing apparatus 100 illustrated in FIG.4 like the first embodiment, and a sensitivity difference correctionprocessing unit 111 of a DSP block 103 (See FIG. 5) has theconfiguration illustrated in FIG. 6.

A Gr mean value calculating unit 121 and a Gb mean value calculatingunit 122 execute processing similar to the processing in the firstembodiment. That is, the Gr mean value calculating unit 121 is a blockto undergo a weighted mean of the pixel values of Gr adjacent to theinterest pixel. The weighted mean mGr of Gr is calculated based on thepreviously described (Expression 7). The Gb mean value calculating unit122 is a block to undergo a weighted mean of the pixel values of Gbadjacent to the interest pixel. The weighted mean mGb of Gb iscalculated based on the previously described expression (Expression 8).

In a sensitivity difference correcting unit 123, the G pixel valueobtained by correcting the sensitivity difference of the interest pixelis calculated according to the above-mentioned (Expression 15) and(Expression 16) by using the weighted mean mGr of Gr obtained by the Grmean value calculating unit 121, the weighted mean mGb of Gb obtained bythe Gb mean value calculating unit 122, and the interest pixel Gb or Gr.

The effect of the corrected G pixel value, which is calculated accordingto the above-mentioned (Expression 15) and (Expression 16) used insecond embodiment, is described by using an image illustrated in FIG. 8.The image illustrated in FIG. 8 is an image where the pixel value of theGb pixel is low and the pixel value of the Gr pixel is high.

Therefore, a difference between the weighted mean mGr of Gr, which iscalculated by the Gr mean value calculating unit 121 according to theabove-mentioned (Expression 7), and the weighted mean mGb of Gb which iscalculated by the Gb mean value calculating unit 122 according to theabove-mentioned (Expression 8), that is, a difference between the mGrand the mGb is increased.

Therefore, as for β((mGr−mGb)/2) and β((mGb−mGr)/2) in (Expression 15)and (Expression 16), in these expressions, the absolute values of((mGr−mGb)/2) and ((mGb−mGr)/2), are increased.

That is, the followings are set:

((mGr−mGb)/2)>TH and ((mGb−mGr)/2)<−TH

or,

((mGr−mGb)/2)<−TH and ((mGb−mGr)/2)>TH

As a result, as shown in the previously described expression (Expression17) and as illustrated in FIG. 9(1), the values of β((mGr−mGb)/2) andβ((mGb−mGr)/2) become 0.

Thus, β(x) becomes 0, and, as a result, the value of the latter half(subsequent to β) of the previously described (Expression 15) and(Expression 16) becomes 0, and the sensitivity difference correctionprocessing will not be performed. With this processing, the pixel valueof the edge portion is output as it is, so that an event in which theedge is blurred does not occur.

On the other hand, when a processing target image portion portion is notan edge portion but a flat portion, a difference between the weightedmean mGr of Gr and the weighted mean mGb of Gb which are calculated bythe previously described (Expression 7) and (Expression 8), that is, adifference between the mGr and the mGb is equal in value to thesensitivity difference which is desired to be corrected. The maximumamount to be corrected is set to a threshold TH in the above-mentioned(Expression 17) considering the image sensor and the opticalcharacteristic.

That is, in connection with β((mGr−mGb)/2) and β((mGb−mGr)/2) in theabove-mentioned (Expression 15) and (Expression 16), in theseexpressions, by using the values (absolute values) of ((mGr−mGb)/2) and((mGb−mGr)/2), the threshold (TH), which is the reference to separatethe flat portion as a correction target and the edge portion which isnot the correction target from each other, is set.

With such a setting, when ((mGr−mGb)/2)>TH and ((mGb−mGr)/2)<−TH, orwhen ((mGr−mGb)/2)<−TH and ((mGb−mGr)/2)>TH, that is, when thefollowings are set, such a portion is determined to be an edge portion.Moreover, as shown in the previously described expression (Expression17) and illustrated in FIG. 9(1), the values of β((mGr−mGb)/2) andβ((mGb−mGr)/2) become 0, and the latter half (subsequent to β) in thepreviously described (Expression 15) and (Expression 16) becomes 0.Accordingly, the sensitivity difference correction processing will notbe performed.

On the other hand, as for ((mGr−mGb)/2)>TH and ((mGb−mGr)/2)<−TH, or((mGr−mGb)/2)<−TH and ((mGb−mGr)/2)>TH, in the cases other than theabove settings, the portion is determined to be a flat portion. As aresult, as shown in the previously described (Expression 17) and asillustrated in FIG. 9(1), the values of β((mGr−mGb)/2) andβ((mGb−mGr)/2) become 1, and the latter half (subsequent to β) in thepreviously described (Expression 15) and (Expression 16) becomes a valueother than 0. That is, the sensitivity difference correction processingwill be performed. In this case, the processing for the flat portionbecomes the processing equivalent to (Expression 11) and (Expression 12)in the embodiments, and the artifact that stands out in the flat portiondue to the sensitivity difference can be corrected.

3. Regarding an Example of Sensitivity Difference Correction Processingin which an Edge Portion is Considered Third Embodiment

Next, an example of sensitivity difference correction processing inwhich an edge portion is considered like the second embodiment isdescribed as a third embodiment of an image processing apparatus of theinvention.

The third embodiment is an embodiment obtained by changing the functionβ(x) which has been described with reference to (Expression 17) and FIG.9(1) in the above-mentioned second embodiment.

In the third embodiment, the function β(x) is set as illustrated in FIG.9(2). That is,

β(x)=1 at −TH1≦x<TH1,

β(x)=0 at x<−TH2 or TH2≦x,

β(x) linearly changes in the range of 0 to 1 at −TH2≦x<−TH1, and

β(x) linearly changes in the range of 1 to 0 at TH1≦x<TH2. However, whenthe condition “−TH2<−TH1<0<TH1<TH2” is set, the function β(x) isapplied.

In this embodiment, although the configuration of the applied imageprocessing apparatus is similar to that of the second embodiment,corrected pixel values are calculated not by using the function β(x)(refer to Expression 17) in the second embodiment but by using afunction β(x) having a definition illustrated in FIG. 9(2) and using thepreviously described (Expression 15) and (Expression 16). As a result,the boundary between the flat portion and the edge portion is smoothed,and the generation of the artifact due to the switching can beprevented.

4. Regarding an Example of Sensitivity Difference Correction Processingin which an Edge Portion is Considered Fourth Embodiment

Next, an example of sensitivity difference correction processing inwhich an edge portion is considered like the second embodiment and thethird embodiment is described as a fourth embodiment of an imageprocessing apparatus of the invention.

The fourth embodiment is an embodiment obtained by changing the functionβ(x) like the third embodiment. In addition, in the present embodiment,(Expression 15) and (Expression 16), which have been applied asexpressions for calculating corrected pixel values in the previouslydescribed second and third embodiment, are not used, but new expressions(Expression 18) and (Expression 19) to be described below are used.

When an interest pixel is Gb, a G pixel value G(-), the sensitivity ofwhich is corrected using the following (Expression 18), is calculated.Moreover, when the interest pixel is Gr, the G pixel value G(-), thesensitivity of which is corrected using the following (Expression 19),is calculated.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 15} \right\rbrack & \; \\{\overset{\_}{G} = {{Gb} + {\gamma \left( \frac{{mGr} - {mGb}}{2} \right)}}} & \left( {{Expression}\mspace{14mu} 18} \right) \\{\overset{\_}{G} = {{Gr} + {\gamma \left( \frac{{mGr} - {mGr}}{2} \right)}}} & \left( {{Expression}\mspace{14mu} 19} \right)\end{matrix}$

A function γ() in the above-mentioned (Expression 18) and (Expression19) is a function defined by the following (Expression 20).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 16} \right\rbrack & \; \\{{\gamma (x)} = \left\{ \begin{matrix}x & {{x} < {TH}} \\0 & {otherwise}\end{matrix} \right.} & \left( {{Expression}\mspace{14mu} 20} \right)\end{matrix}$

In the above-mentioned (Expression 20), TH is a threshold parameter toseparate a flat portion and an edge portion from each other. As afunction γ(x) defined in the above-mentioned (Expression 20), variousfunctions may be used. However, the function illustrated in FIG. 10(1)is used for example. That is, γ(x)=0 at x<−TH or TH≦x, and γ(x) linearlychanges in the range of −1 to 1 at −TH≦x<TH. The function γ(x) with sucha setting is used.

In the present embodiment, although the configuration of the appliedimage processing apparatus is similar to the configuration applied inthe second embodiment, corrected pixel values are calculated not byusing (Expression 15) to (Expression 17) used in the second embodimentbut by applying the above-mentioned (Expression 20), that is, byapplying the function γ(x) having a definition illustrated in FIG. 10(1)and by applying the previously described (Expression 18) and (Expression19).

5. Regarding an Example of Sensitivity Difference Correction Processingin which an Edge Portion is Considered Fifth Embodiment

Next, an example of sensitivity difference correction processing inwhich an edge portion is considered like the second embodiment to thefourth embodiment is described as a fifth embodiment of an imageprocessing apparatus of the invention.

The fifth embodiment is an embodiment obtained by changing the functionγ(x) in the fourth embodiment which has been described with reference to(Expression 20) and FIG. 10(1).

In the fifth embodiment, the function γ(x) is set as illustrated in FIG.10(2). That is,

γ(x)=0 at x<−TH3 or TH3≦x, and

γ(x)=1 at −TH2≦x<TH1 or TH1≦x<TH2. Moreover, γ(x) linearly changes inthe range of 0 to −1 at −TH3≦x<−TH2, γ(x) linearly changes in the rangeof −1 to 1 at −TH1≦x<TH1, and γ(x) linearly changes in the range of 1 to0 at TH2≦x<TH3. However, the function γ(x) with the setting of−TH3<−TH2<−TH1<0<TH1<TH2<TH3 is applied.

In the present embodiment, although the configuration of the appliedimage processing apparatus is similar to that of the fourth embodiment,corrected pixel values are calculated not by using the function γ(x)(see FIG. 10(1)) in the fourth embodiment but by using the function γ(x)having a definition illustrated in FIG. 10(2) and using the previouslydescribed (Expression 18) and (Expression 19). As a result, the boundarybetween the flat portion and the edge portion is smoothed and thegeneration of the artifact due to the switching can be prevented.

The calculation process of (Expression 15) and (Expression 16), which isused as the expression for calculating the corrected pixel value in thesecond and third embodiments, requires multiplications. However,(Expression 18) and (Expression 19), which is used as the expression forcalculating the corrected pixel value in the fourth and fifthembodiments, has a characteristic that additions and conditionalbranching process are necessary but multiplications are not necessary.Accordingly, any of the expressions may be suitably selected consideringthe packaging structure.

6. Regarding an Embodiment (Sixth Embodiment) in which SensitivityDifference Correction Processing and Demosaic Processing are Combined

Next, an embodiment (sixth embodiment) that combines sensitivitydifference correction processing and demosaic processing is described asa sixth embodiment of an image processing apparatus of the invention.

The image processing apparatus of the sixth embodiment has theconfiguration of the image processing apparatus illustrated in FIG. 4for example, like in the previously described first to fifthembodiments. In an imaging apparatus 100 illustrated in FIG. 4, lightincident through an optical lens 101 is incident on an imaging device102 configured by a CCD, a CMOS image sensor, or the like, and imagedata is output. The output image data is input to a DSP block 103, andis then subjected to signal processing for an output signal here. Afterthat, output image 104 is output.

An example of the configuration of the DSP block 103 in the presentembodiment is described with reference to FIG. 11. An image of the Bayerpattern illustrated in FIG. 1(1) is output from the imaging device 102illustrated in FIG. 4 and is input to the DSP block 103. However, thisinput data has the sensitivity differences according to the positions ofG. That is, the input data becomes mosaic data that corresponds to thearray (R, Gb, Gr, B) illustrated in FIG. 1(2).

The DSP block 103 in the present embodiment includes a white balanceprocessing unit 112, a sensitivity difference correction and demosaicprocessing unit 131, a gamma correcting unit 114, and a YC convertingunit 115 as illustrated in FIG. 11.

The white balance processing unit 112, the gamma correcting unit 114,and the YC converting unit 115 have the same configurations and executesthe same processing as the configurations and the processing which havebeen described with reference to FIG. 5 in the first embodiment.

In the sensitivity difference correction and demosaic processing unit131, the correction processing of a sensitivity difference between a Gbpixel and a Gr pixel and the demosaic processing are performed, so thatan image of three channels (R, G, and B) is output.

The sensitivity difference correction and demosaic processing unit 131is described with reference to FIG. 12. A line memory 141 is used toacquire pixel values of vertically adjacent pixels of an interest pixel.The imaging device 102 of the imaging apparatus 100 illustrated in FIG.4 sequentially outputs the image in the order of so-called raster scanthat is illustrated in FIG. 13. The line memory 141 temporarily storespixel value data corresponding to a plurality of scan lines. At the timeof performing the sensitivity difference correction processing, thepixel value data stored in the line memory 141 is used, for example,when making reference to the vertically adjacent pixels.

A Gr mean calculating unit 142 is a block to undergo a weighted mean ofthe pixel values of Gr adjacent to an interest pixel. When the weightedmean of Gr is assumed to be mGr, the mGr is calculated based on thepreviously described (Expression 7).

A Gb mean value calculating unit 142 is a block to undergo a weightedmean of the pixel values of Gb adjacent to the interest pixel. When theweighted mean of Gb is assumed to be mGb, the mGb can be calculatedbased on the previously described (Expression 8).

In an R mean calculating unit 144, a mean value mR of R adjacent to theinterest pixel is calculated by the following (Expression 21).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 17} \right\rbrack & \; \\{{mR} = \frac{\sum\limits_{k}^{\;}\left( {R_{k} \times C_{k}} \right)}{\sum\limits_{k}^{\;}C_{k}}} & \left( {{Expression}\mspace{14mu} 21} \right)\end{matrix}$

In the above (Expression 21), k represents a pixel number of a certainsurrounding pixel, Rk represents a pixel value of an R pixelcorresponding to the number, and Ck represents a weighting factorcorresponding to the distance from a center pixel.

In a B mean calculating unit 145, a mean value mB of B pixels adjacentto the interest pixel is calculated based on the following (Expression22).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 18} \right\rbrack & \; \\{{mB} = \frac{\sum\limits_{l}^{\;}\left( {B_{l} \times C_{l}} \right)}{\sum\limits_{l}^{\;}C_{l}}} & \left( {{Expression}\mspace{14mu} 22} \right)\end{matrix}$

In the above (Expression 21), 1 represents a pixel number of a certainsurrounding pixel, B1 represents a pixel value of a B pixelcorresponding to the number, and C1 represents a weighting factorcorresponding to the distance from the center pixel.

In a direction determination processing unit 146, the absolute value ofLaplacian or gradient for the horizontal direction of the interest pixeland the absolute value of Laplacian or gradient for the verticaldirection of the interest pixel are compared with each other and, as aresult, the direction of the edge around the interest pixel is estimatedand output. An example of calculating the gradient as an edge estimationvalue is described with reference to FIG. 14. The edge estimation valuein the horizontal direction and the edge estimation value in thevertical direction can be calculated according to the following(Expression 23) and (Expression 24) respectively.

[Formula 19]

Eh=|G ₂ −G ₃|  (Expression 23)

Ev=|G ₁ −G ₄|  (Expression 24)

G1, G2, G3, and G4 in the above-mentioned (Expression 23) and(Expression 24) are G pixels illustrated in FIG. 14. The edge in thehorizontal direction is detected by (Expression 23) and the edge of thevertical direction is calculated by (Expression 24). The above-mentioned(Expression 23) and (Expression 24) may produce the sum of the absolutevalues of Laplacian or gradient of surrounding pixels.

As for Eh and Ev calculated in the above-mentioned (Expression 23) and(Expression 24), when the relation “Eh<Ev” is established, the relationmeans that there is a stronger edge in the vertical direction and acorrelation with the horizontal direction is strong.

As for the Eh and Ev calculated in the above-mentioned (Expression 23)and (Expression 24), when a relation “Ev<Eh” is established, therelation means that there is a stronger edge in the horizontal directionand a correlation with the vertical direction is strong.

In a sensitivity difference correcting unit 147, a sensitivitydifference between the Gb pixel and the Gr pixel is corrected. Any oneof techniques described in the previously described first to fifthembodiments can be used as the correction technique. Specifically, acorrected pixel value of a G pixel is calculated by any one of theoperations of (Expression 11) and (Expression 12) described in the firstembodiment, (Expression 15) and (Expression 16) described in the secondand third embodiments, and (Expression 18) and (Expression 19) describedin the fourth and fifth embodiments.

In a G mean calculating unit 148, a mean value mG of G pixels iscalculated based on the following expression (Expression 25) by using aweighted mean mGr of Gr, which is calculated by the Gr mean calculatingunit 142 according to the previously described (Expression 7), and aweighted mean mGb of Gb which is calculated by the Gb mean calculatingunit 143 according to the previously described (Expression 8).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 20} \right\rbrack & \; \\{{mG} = \frac{{mGr} + {mGb}}{2}} & \left( {{Expression}\mspace{14mu} 25} \right)\end{matrix}$

A correlation processing unit 149 generates and outputs a three-channel(R, G, and B) image of the Bayer pattern signal by performingcorrelation processing by using output values of the respective meancalculating units 142 to 145 and 148, the direction determinationprocessing unit 146, and the sensitivity difference correcting unit 147.

An example of processing executed in the correlation processing unit 149is described referring to FIG. 15 and the subsequent figures. First, thecorrelation processing unit 149 performs pixel interpolation processingof calculating a pixel value of a G pixel at the position of an R pixelor a B pixel in a G high frequency generating unit 151 illustrated inFIG. 15. The G pixel value at the position of the G pixel is applied asit is. Hereinbelow, a description is sequentially made about (1) G pixelvalue calculation processing for positions of R and B pixels and (2) Gpixel value calculation processing for a position of a G pixel.

(1) G Pixel Value Calculation Processing for Positions of R and B Pixels

First, G pixel value calculation processing for positions of R and Bpixels is described. As for the edge estimation values Eh and Ev of thehorizontal direction and the vertical direction which have beenpreviously described referring to (Expression 23) and (Expression 24),when the relation “Eh<Ev” is established, that is, when it is determinedthat there is a stronger edge in the vertical direction and acorrelation with the horizontal direction is strong, the estimationpixel values of G pixels at the positions of the R and B pixels arecalculated by using the following pixel value calculation expression(Expression 26).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 21} \right\rbrack & \; \\{\overset{\Cap}{G} = \frac{G_{2} + G_{3}}{2}} & \left( {{Expression}\mspace{14mu} 26} \right)\end{matrix}$

As illustrated in FIG. 16(A), G1, G2, G3, and G4 are G pixels adjacentto an interest pixel. The above expression is an expression to calculatean estimated pixel value of a G pixel at the position of the R pixel orthe B pixel by using a G pixel value in the horizontal direction. Thatis, demosaic processing of determining a pixel value of a processingtarget pixel is performed by setting contribution of a pixel value of areference pixel in a direction where a change in the pixel value issmall to a high level.

On the other hand, when the relation “Eh>Ev” is established, that is,when there is a stronger edge in the horizontal direction and it isdetermined that a correlation with the vertical direction is strong, thepixel value of the G pixel is calculated by using the following pixelvalue calculation expression (Expression 27).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 22} \right\rbrack & \; \\{\overset{\Cap}{G} = \frac{G_{1} + G_{4}}{2}} & \left( {{Expression}\mspace{14mu} 27} \right)\end{matrix}$

As illustrated in FIG. 16(A), G1, G2, G3, and G4 are G pixels adjacentto the interest pixel. The above-mentioned expression is an expressionto calculate an estimated pixel value of the G pixel at the position ofthe R pixel or the B pixel by using the G pixel value of the horizontaldirection. That is, demosaic processing of determining the pixel valueof the processing target pixel is performed by setting contribution ofthe pixel value of the reference pixel in a direction where a change inthe pixel value is small to a high level.

(2) G Pixel Value Calculation Processing for a Position of a G Pixel

Next, G pixel value calculation processing for a position of a G pixelis described. When a center pixel (interest pixel) is a G pixel asillustrated in FIG. 16(B) the following (Expression 28) is applied andthe pixel value of the G pixel is used as it is.

[Formula 23]

{hacek over (G)}=G₅  (Expression 28)

When the center pixel (interest pixel) is a G pixel as illustrated inFIG. 16(B), the pixel value is used as it is.

G pixels at all pixel positions are obtained based on theabove-mentioned (Expression 26), (Expression 27), and (Expression 28).As for the G pixel value used at the time of performing the demosaicprocessing, it is desirable to use the G pixel values correctedaccording to each embodiment described above as pixel values of the Grpixel and the Gb pixel.

An R/B high frequency generating unit 152 calculates an R pixel and a Bpixel for each position of all pixels by using the G pixel, a mean valuemR of R adjacent to the interest pixel which is calculated based on(Expression 21) by the R mean calculating unit 144, and a mean value mBof B adjacent to the interest pixel which is calculated based on(Expression 22) by the B mean calculating unit 145. That is, theinterpolation processing of the R pixels and the B pixels is executed.

[Formula 24]

{circumflex over (R)}=(mR−mG)+{hacek over (G)}  (Expression 29)

{circumflex over (B)}=(mB−mG)+{hacek over (G)}  (Expression 30)

The mR, mG, and mB in the above (Expression 29) and (Expression 30) areweighted mean values of surrounding pixels of the interest pixel whichare calculated according to the above-mentioned (Expression 21),(Expression 22), and (Expression 25) respectively.

In the above-mentioned (Expression 29) and (Expression 30), (mR−mG) and(mB−mG) mean offsets of low frequency components of respective colorchannels. That is, the correlation processing means processing whichuses the fact that the correlation between a high frequency component inthe G pixel and a high frequency component in the R pixel or the B pixelis strong.

As for the demosaic processing, the processing may be executed by usingexisting processing. However, in the processing of the invention, theprocessing is performed by applying the corrected pixel value G. Forexample, examples of the processing are disclosed in Japanese Patent No.2931520 or the document “Soo-Chang Pei, et al, Effective ColorInterpolation in CCD Color Filter Arrays Using Signal Correlation, IEEEtrans. On Circuits and Systems for Video Technology, Vol 13-6 (2003)”.However, in the processing of the invention, the processing is performedby applying the corrected pixel value G described above.

As disclosed in the above documents, in order to achieve the demosaicprocessing with a DSP, a memory to store pixel values of adjacentpixels, for example, the line memory 141; the mean calculating units144, 145, and 147 for respective colors (respective R, G, and B colorsin the present embodiment); and the correlation processing unit 148 maybe used. Moreover, in order to achieve the processing of correcting thesensitivity difference between the Gb pixel and the Gr pixel with a DSP,a memory to store pixel values of adjacent pixels, for example, the linememory 141; the Gr mean calculating unit 142; the Gb mean calculatingunit 143; and the sensitivity difference correcting unit 146 may beused. That is, there may be a common memory and a common processing unitused for the sensitivity difference correction processing and thedemosaic processing. Therefore, components necessary for both theprocessing may be integrated as illustrated in FIG. 12.

Therefore, by realizing a configuration in which the operationsdescribed in the above embodiments are executable, with use of hardwarewhich executes existing demosaic processing, a configuration in whichthe sensitivity difference correction processing as well as the demosaicprocessing is executed is achieved.

7. Regarding an Embodiment (Seventh Embodiment) in which SensitivityDifference Correction Processing and Demosaic Processing are Combined

Next, modified processing of the embodiment (the sixth embodiment), inwhich the sensitivity difference correction processing and the demosaicprocessing are combined, is described as a seventh embodiment of animage processing apparatus of the invention.

The image processing apparatus of the seventh embodiment has theconfiguration of the image processing apparatus, for example,illustrated in FIG. 4 like the previously described first to fifthembodiments. In an imaging apparatus 100 illustrated in FIG. 4, lightincident through an optical lens 101 is incident on an imaging device102 configured by a CCD, a CMOS image sensor, or the like, and imagedata is output. The output image data is input to a DSP block 103, andsignal processing for an output signal is performed here. After that,output image 104 is output.

A configuration example of the DSP block 103 in the present embodimenthas the configuration which has been described with reference to FIG. 11like the above-described sixth embodiment.

A sensitivity difference correction and demosaic processing unit 131 ofthe present embodiment is described with reference to FIG. 17.

A line memory 141 is used to acquire pixel values of vertically adjacentpixels of an interest pixel. A Gr mean calculating unit 142 is a blockto undergo a weighted mean of the pixel values of Gr adjacent to theinterest pixel. When the weighted mean of Gr is assumed to be mGr, themGr is calculated based on the previously described (Expression 7). A Gbmean value calculating unit 142 is a block to undergo a weighted mean ofthe pixel values of Gb adjacent to the interest pixel. When the weightedmean of Gb is assumed to be mGb, the mGb can be calculated based on thepreviously described (Expression 8).

In an R mean calculating unit 144, a mean value mR of R adjacent to theinterest pixel is calculated based on the previously described(Expression 21). In a B mean calculating unit 145, a mean value mB of Bpixels adjacent to the interest pixel is calculated based on thepreviously described (Expression 22).

In a direction determination processing unit 146, edge estimation valuesin a horizontal direction and a vertical direction are calculatedaccording to the previously described (Expression 23) and (Expression24). The direction determination processing unit 146 calculates ahorizontal-direction edge estimation value Eh and a vertical-directionedge estimation value Ev.

In an edge adding unit 161, edge component information which does notdepend on the direction is calculated according to the following(Expression 31).

Ehv=Eh+Ev  (Expression 31)

Edge components that do not depend on the direction are obtained basedon the above-mentioned (Expression 31). The value (edge componentinformation) Ehv calculated based on the above-mentioned (Expression 31)has the following characteristic. The Ehv increases in an edge portion,and decreases in a flat portion. That is, the value has such acharacteristic.

In the configuration that calculates the G pixel values as the correctedpixel values of a Gr pixel and a Gb pixel, by applying a function β(x)that produces a value in the range of 1 to 0 according to whether it isthe edge portion, which is described in the previously described secondto fourth embodiments, weighted mean values mGr and mGb of adjacent Grand Gb pixels of the interest pixel, which are obtained based on(Expression 7) and (Expression 8), are used in the previously described(Expression 15) to (Expression 20). The processing described in thesecond to fourth embodiments is a method which does not deteriorate theedge portion while removing the artifact that appears in the flatportion due to the difference in the sensitivity between the Gb pixeland the Gr pixel.

Although the function β(x) that produces a value in the range of 1 to 0according to whether it is an edge portion or not in the second tofourth embodiments, most suitable processing according to thedetermination result can be realized by determining whether it is theedge portion or the flat portion even by using the Ehv which iscalculated according to the above-mentioned (Expression 31) instead ofthe function β(x).

Specifically, for example, the following (Expression 32) can be usedinstead of the function β(x) of (Expression 17) which is described inthe second embodiment.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 25} \right\rbrack & \; \\{{\beta (x)} = \left\{ \begin{matrix}1 & {{Ehv} < {TH}} \\0 & {otherwise}\end{matrix} \right.} & \left( {{Expression}\mspace{14mu} 32} \right)\end{matrix}$

In the above-mentioned (Expression 32), TH is a threshold parameter usedto separate the flat portion and the edge portion from each other. Whenedge component information (Ehv), which does not depend on the directionand is calculated based on the above-mentioned (Expression 31), is lessthan the preset threshold TH, β(x)=1. On the other hand, when the edgecomponent information (Ehv) is the preset threshold TH or more, β(x)=0.

Thus, the invention may provide a configuration that calculatescorrected pixel values of G pixels by using the edge componentinformation (Ehv), which does not depend on the direction and iscalculated based on the above-mentioned (Expression 31), and by using(Expression 15) and (Expression 16) described in the second embodiment.Even for the function β(x) applied in the third and fourth embodiments,the above-mentioned (Expression 32) which uses the edge elementinformation (Ehv), which does not depend on the direction and iscalculated based on the above-mentioned (Expression 31), can be used.

According to the processing of the invention, it is possible to removethe artifact which especially appears in the flat portion of an image bycorrecting the sensitivity difference that exists between the colorfilters having the same spectral characteristic. One example that showsthe effect according to the invention is described with reference toFIG. 18. In FIG. 18, (A) an image without having been subjected to Gpixel correction processing according to the invention and (B) an imagehaving been subjected to the G pixel correction processing according tothe invention are illustrated in parallel.

Although it is difficult to confirm because the figure illustrates animage, which is originally a color image, in gray scale, the artifact inthe check appears in the image (A) which has not been subjected to the Gpixel correction processing and it is observed that the artifact isreduced in the image (B) which has been subjected to the G pixelcorrection processing according to the invention.

Thus, with use of the configuration of the invention, it becomespossible to correct the sensitivity difference between the color filtershaving the same spectral characteristic with a very little amount ofoperations. Moreover, the pixel value correction processing of theinvention has a configuration similar to the configuration of existingdemosaic processing, the operation and memory are shared when the pixelvalue correction processing of the invention is integrated with theexisting demosaic processing, and hardware may be built with a smalleramount of operation and memory capacity.

8. Regarding a Hardware Configuration Example of an Image ProcessingApparatus

Lastly, a specific hardware configuration of one image processingapparatus of the invention is described with reference to FIG. 19. FIG.19 illustrates an example of a personal computer (PC) as an example ofthe image processing apparatus of the invention. However, as describedabove, the image processing apparatus of the invention may be realizedby any of various apparatuses such as an imaging apparatus, a TV, and aplayer as well as the PC as long as the apparatus receives a mosaicimage, which is imaged by applying a specific color filter to a singleplate-type image sensor and can perform the demosaic processing.

A Central Processing Unit (CPU) 501 executes various processingaccording to a program stored in a Read Only Memory (ROM) 502 or astorage unit 508. For example, the CPU executes brightness controlprocessing described in each of the embodiments described above.Programs executed by the CPU 501, data, etc. are properly stored in aRandom Access Memory (RAM) 503. The CPU 501, the ROM 502, and the RAM503 are connected to one another by a bus 504.

The CPU 501 is connected to an input/output interface 505 via the bus504. The input/output interface 505 is connected to an input unit 506configured by a keyboard, a mouse, a microphone, etc; and an output unit507 configured by a display, a speaker, etc. Moreover, processing targetdata, for example, the mosaic image (the photoelectric conversionsignal) imaged by applying a specific color filter to a singleplate-type image sensor is input through the input unit 506.

The CPU 501 executes various processing corresponding to instructionsinput from the input unit 506 and outputs processing results, forexample, to the output unit 507. The output unit 507 is configured by adisplay, a speaker, etc., and the image generated by the above-describedprocessing is displayed on the display.

The storage unit 508 connected to the input/output interface 505 isconfigured, for example, by a hard disk, and stores programs to beexecuted by the CPU 501, and various kinds of data. The communicationunit 509 communicates with an external device through a network such asthe Internet and a local area network.

The drive 510 connected to the input/output interface 505 drives aremovable medium 511 such as a magnetic disc, an optical disc, a magnetooptical disc, and a semiconductor memory and acquires recorded programs,data, etc. The acquired program and data are transmitted to and storedin the storage unit 508 as necessary.

Hereinabove, the invention has been described in detail with referenceto a specific example. However, it is apparent that those skilled in theart may make changes or substitutions to the embodiments withoutdeparting from the spirit of the invention. That is, the invention hasbeen disclosed in the form of illustrations, and the invention shouldnot be interpreted limitedly. Reference should be made to the appendedclaims, in order to assess the scope of the invention.

Moreover, a series of processing which has been described in thespecification can be executed with a configuration of hardware,software, or a combination of both. When processing based on software isexecuted, the processing may be executed by installing a program, whichis a record of a process sequence, in a memory inside a computer whichis built in dedicated hardware, or by installing a program in a generalpurpose computer in which various kinds of processing can be performed.For example, the program may be recorded in a recording mediumbeforehand. Besides the installation from a recording medium to acomputer, the program may be received through a network such as a LocalArea Network (LAN) and is then installed in a recording medium such as abuilt-in hard disc.

Various processing described in the specification may not only beexecuted in time series manner according to the order of description,but also be executed in parallel or individually according to processingcapacity of an apparatus which executes the processing or as necessary.The term “system” in this specification means a configuration of alogical set of a plurality of devices, but the each device is notnecessarily provided in the same casing.

INDUSTRIAL APPLICABILITY

As described above, according to a configuration of one example of theinvention, provided are a device and a method that execute sensitivitycorrection processing on an image signal generated by a singleplate-type image sensor through a color filter. Specifically, thesensitivity difference correction of Gr and Gb signals included in animage signal, for example, an RGB signal, generated by a singleplate-type image sensor through a color filter is executed. A pixelvalue of a color filter unit which has the same color as a correctiontarget pixel and is provided in surroundings of the correction targetpixel is acquired. A difference between weighted mean pixel values “a”and “b” of two kinds of pixel groups “A” and “B” classified according topositions of pixels is added to a pixel value of the correction targetpixel to calculate an additional value. A mean value of the pixel valueof the correction target pixel and the additional value is calculated asa corrected pixel value of the correction target pixel. According to theprocessing of the present invention, correction processing can beperformed with a simple operation and with high accuracy.

REFERENCE SIGNS LIST

-   100 Imaging apparatus (Image processing apparatus)-   101 Optical lens-   102 Imaging device-   103 DSP block-   104 Output image-   111 Sensitivity difference correction processing unit-   112 White balance processing unit-   113 Demosaic processing unit-   114 Gamma correcting unit-   115 YC converting unit-   121 Gr mean value calculating unit-   122 Gb mean value calculating unit-   123 Sensitivity difference correcting unit-   131 White balance processing unit-   141 Line memory-   142 Gr mean value calculating unit-   143 Gb mean value calculating unit-   144 R mean value calculating unit-   145 B mean value calculating unit-   146 Direction determination processing unit-   147 Sensitivity difference correcting unit-   148 G mean calculating unit-   149 Correlation processing unit-   151 G high frequency generating unit-   152 R/B high frequency generating unit-   161 Edge adding unit-   162 Sensitivity difference correcting unit-   501 CPU-   502 ROM-   503 RAM-   504 Bus-   505 Input/output interface-   506 Input unit-   507 Output unit-   508 Storage unit-   509 Communication unit-   510 Drive-   511 Removable medium

1.-10. (canceled)
 11. An image processing apparatus comprising: asensitivity difference correction processing unit configured to receivean image signal and executes sensitivity difference correctionprocessing and demosaic processing.
 12. The image processing apparatusaccording to claim 11, wherein the color filter is arrayed in a Bayerpattern, and the sensitivity difference correction processing unit has aconfiguration which a sensitivity difference between a Gr pixel and a Gbpixel included in an image which is imaged through the color filter ofthe Bayer pattern is corrected.
 13. The image processing apparatusaccording to claim 12, wherein the sensitivity difference correctionprocessing unit: calculates, when performing correction processing onthe Gr pixel and the Gb pixel, weighted pixel value means mGr and mGbcorresponding to distances of the Gr pixel and the Gb pixel, which areadjacent to the correction target pixel, from the correction targetpixel; calculates an additional value by adding a difference between themGr and the mGb to the pixel value of the correction target pixel; andcalculates the mean value of the pixel value of the correction targetpixel and the additional value as the corrected pixel value of thecorrection target pixel.
 14. The image processing apparatus according toclaim 11, wherein the sensitivity difference correction processing unit:calculates a final corrected pixel value by reducing a contributionlevel of the additional value when an adjacent image area including thecorrection target pixel is an edge area; and calculates the finalcorrected pixel value by increasing the contribution level of theadditional value when the adjacent image area including the correctiontarget pixel is a flat area.
 15. The image processing apparatusaccording to claim 11, wherein the image signal being output by an imagesensor based on incident light being incident through a color filter;and wherein the sensitivity difference positive processing unit:acquires a pixel value of a color filter unit which has a same color asa correction target pixel and is present in surroundings of thecorrection target pixel; calculates an additional value by adding adifference between weighted mean pixel values “a” and “b” of two kindsof pixel groups “A” and “B” classified according to positions of thepixels to the pixel value of the correction target pixel, the weightedmean pixel values corresponding to distances of the pixel groups fromthe correction target pixel; and calculates a mean value of the pixelvalue of the correction target pixel and the additional value as acorrected pixel value of the correction target pixel.
 16. The imageprocessing apparatus according to claim 15, wherein the sensitivitydifference correction processing unit has a configuration in whichdemosaic processing of determining a pixel value of a processing targetpixel is performed based on a pixel value of a reference pixel bydetermining the contribution level of the pixel value of the referencepixel adjacent to the processing target pixel according to a result ofthe edge direction determination of an image area including theprocessing target pixel of the demosaic processing.
 17. The imageprocessing apparatus according to claim 16, wherein the sensitivitydifference correction processing unit has a configuration in whichdemosaic processing of determining the pixel value of the processingtarget pixel is performed by setting the contribution level of the pixelvalue of the reference pixel in a direction where a change in pixelvalue is small to a high level, according to a result of the edgedirection determination of the image area including the processingtarget pixel of the demosaic processing.
 18. A system comprising: animaging device; and the sensitivity difference correction processingunit according to any one of claims 11 to 17.